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SUMMARY 


Background 

The  human  resources  requirements  for  new  Air  Force  systems  repre¬ 
sent  a  substantial  portion  of  system  ownership  costs.  These  require¬ 
ments  should,  therefore,  be  assessed  early  in  system  design  stages  to 
ensure  optimal  use  and  lowest  possible  costs  of  human  resources.  Appro¬ 
priate  data,  however,  are  not  often  available  in  the  early  stages  of  the 
system  design  process.  Recent  studies  have  suggested  that  estimates  of 
maintenance,  manpower,  and  training  requirements  for  Air  Force  systems 
in  the  early  design  stages  could  be  made  by  technicians  with  maintenance 
experience  on  similar,  currently  operational  systems. 

Obj  ectives 

The  objectives  of  the  study  were  to  exercise  an  expert  estimate 
method  of  generating  maintenance  and  manpower  data  for  new  systems,  to 
determine  the  accuracy  of  this  estimated  data,  to  collect  cost  data  for 
implementing  the  method,  to  evaluate  a  prototype  users  guide  describing 
the  method,  and  to  prepare  a  revised  users  guide  for  the  expert  estimate 
method. 

Approach 

Using  only  an  early  design  phase  description  of  a  ground-based 
radar  system,  estimates  were  made  by  maintenance  technicians  of  mainte¬ 
nance,  manpower,  and  training  requirements  for  that  system.  These 
estimates  were  made  following  the  procedures  and  recommendations  con¬ 
tained  in  a  prototype  users  guide  for  collecting  expert  estimates.  The 
accuracy  of  the  estimates  was  determined  by  comparing  the  estimated  data 
with  operational  data  obtained  either  from  Air  Force  maintenance  records 
or  from  maintenance  experts  working  with  the  radar  system. 

Based  on  the  recommendations  in  the  prototype  guide,  an  early 
design  phase  engineering  description  of  the  AN/TPS-43(E)  radar  system 
was  prepared,  consisting  of  a  basic  engineering  description  package  and 
a  supplement  to  the  basic  package.  A  questionnaire  was  designed  to 
collect  estimates  of  maintenance  task  times,  crew  size,  skill  level, 
percent  of  maintenance  task  occurrences,  troubleshooting  difficulty, 
career  field/Air  Force  Specialty  Code  (AFSC)  requirements,  training 
times  and  support  equipment  requirements.  Seventy  radar  maintenance 
technicians  from  the  303X1  ASFC  (Air  Traffic  Control  Radar),  the  303X2 
AFSC  (Aircraft  Control  and  Warning  Radar),  and  the  303X3  AFSC  (Automatic 
Tracking  Radar)  with  varying  levels  of  maintenance  experience  partici¬ 
pated  as  expert  estimators.  The  technicians  were  located  at  five  Air 
Force  bases.  The  technicians  provided  their  maintenance  background 
data,  rated  their  confidence  in  their  estimates,  indicated  the  relative 
importance  of  the  data  sources  upon  which  they  based  their  estimates, 
and  indicated  the  effect  on  their  estimates  of  the  supplemental  engi¬ 
neering  data. 
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The  thrust  of  the  analysis  was  to  determine  (a)  the  accuracy  of 
the  estimates,  (b)  how  the  estimated  data  were  affected  by  qualifica¬ 
tions  of  estimators,  (c)  the  amount  of  engineering  detail  which  should 
be  presented  in  the  engineering  description  package,  (d)  the  kinds  of 
human  resources  data  which  can  be  estimated,  (e)  the  types  of  informa¬ 
tion  that  technicians  used  to  make  their  estimates,  and  (f)  the  confi¬ 
dence  technicians  placed  in  their  estimates.  Cost  data  for  applying 
the  technique  were  tracked  and  examined.  Critical  evaluations  of  the 
prototype  guide  itself  were  obtained  from  Air  Force  manpower  profes¬ 
sionals.  Findings  from  all  phases  of  the  effort  were  incorporated  in 
a  revised  users  guide  for  the  expert  estimate  method. 

Results  and  Conclusions 


A  comparison  of  the  estimates  with  the  operational  data  indicates 
that  crew  size,  skill  level,  troubleshooting  difficulty,  career  field/ 
AFSC  requirements,  and  training  times  can  be  satisfactorily  estimated. 
Acceptable  derived  estimates  were  produced  for  maintenance  person- 
hours.  The  data  items  that  were  not  satisfactorily  estimated  were 
percentage  of  maintenance  task  occurrences,  support  equipment  require¬ 
ments,  and  derived  person-hour  estimates  for  troubleshooting  and 
specific  off-equipment  maintenance  tasks. 

Recommendations  in  the  prototype  users  guide  regarding  the  form 
and  content  of  the  engineering  description  package  and  the  form  and 
content  of  the  data  collection  questionnaire  were  supported.  Recom¬ 
mendations  in  the  users  guide  regarding  the  qualifications  of  the 
estimators  were  generally  supported.  Technician  estimators  should 
hold  an  AFSC  which  is  the  same  as  or  similar  to  the  AFSC  proposed  for 
the  new  system.  All  estimators  should  have  attained  a  skill  level  of 
at  least  5  in  their  AFSC.  However,  the  cost  data  indicated  that 
increasing  the  recommended  10  estimators  to  25  incurred  only  a  small 
percentage  increase  in  overall  person-hour  costs. 

Estimators  reported  that  they  were  slightly  less  than  "Confident" 
about  their  estimates,  that  any  given  estimate  was  based  more  on 
their  maintenance  experience  (67  percent)  than  on  the  engineering  data 
provided  (33  percent),  and  that  the  supplemental  engineering  data  had 
no  effect  on  their  estimates. 

Cost  data  reflected  that  the  number  of  person-hours  required  for 
an  effort  similar  in  scope  to  the  current  study  would  be  657.5  person- 
hours.  Travel  and  material  costs  were  recognized  but  not  estimated 
because  of  the  potential  variations  in  travel  and  material  requirements 
and  costs.  No  similar  data-generating  methods  could  be  identified  for 
cost-comparison  purposes. 

Manpower  professionals  evaluated  the  prototype  guide  in  terms  of 
clarity  and  content.  These  evaluation  comments,  along  with  the  findings 
of  the  application  of  the  expert  estimate  method  to  the  radar  system, 
were  the  bases  upon  which  revisions  were  made  to  the  guide. 
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INTRODUCTION 


Background 

Increasing  costs  of  new  Air  Force  weapons  systems  and  limited 
budgets  represent  major  constraints  for  Air  Force  planners  and 
designers.  Research  programs  initiated  in  response  to  these  con¬ 
straints  have  resulted  in  new  or  improved  materials,  more  efficient 
designs,  and  new  manufacturing  techniques  and  processes. 

Research  has  also  been  initiated  to  investigate  the  impact  of  new 
systems  on  human  resource  requirements.  The  assumptions  underlying 
this  area  of  research  are  that  while  advanced  materials  and  designs 
help  reduce  costs,  new  systems  must  also  efficiently  utilize  personnel 
and  provide  for  effective  levels  of  personnel  performance.  Further, 
if  a  new  system  appears  to  have  an  adverse  impact  on  human  resource 
requirements,  the  earlier  in  the  design  process  this  impact  can  be 
identified,  the  easier  it  is  to  effect  changes  in  the  system  design  to 
overcome  this  adverse  effect. 

The  problem  of  gathering  human  resource  data  in  the  early  stages 
of  system  design  is  gaining  increasing  attention.  Two  recent  studies 
(Sauer  &  Askren,  1978a;  Whalen  &  Askren,  1974)  have  investigated  the 
feasibility  of  obtaining  maintenance  personnel  requirements  and  per¬ 
formance  data  during  the  conceptual  phase  of  system  design  using 
subjective  estimate  techniques.  These  data  were  obtained  through 
estimates  made  by  experienced  maintenance  technicians.  This  approach, 
called  the  expert  estimate  method,  produced  satisfactory  estimates  of 
maintenance  person-hours,  crew  size,  skill  level,  career  field,  and 
task  difficulty.  On  the  basis  of  these  results,  Sauer  &  Askren 
(1978b)  also  prepared  a  prototype  users  guide  describing  the  steps 
and  procedures  for  collecting  expert  estimates. 

While  the  results  of  these  studies  were  encouraging,  it  should  be 
noted  that  they  were  limited  to  a  small  number  of  aircraft  avionics 
subsystems  and  used  only  aircraft  avionics  technicians  as  estimators. 
Additional  research  is  necessary  to  provide  a  better  understanding  of 
the  general  utility  of  the  expert  estimate  method  of  generating  per¬ 
sonnel  requirements  and  performance  data.  The  method,  as  described  in 
the  prototype  users  guide,  should  be  extended  to  a  different  technol¬ 
ogy  area  and  technician  specialty  field.  In  this  way,  it  is  possible 
to  assess  how  well  the  procedure  transfers  to  other  systems  and  to 
evaluate  the  procedures  contained  in  the  prototype  users  guide. 

Objectives 

The  objectives  of  the  study  were  to  exercise  an  expert  estimate 
method  of  generating  maintenance  and  manpower  data  for  new  systems,  to 


determine  the  accuracy  of  these  estimated  data,  to  collect  cost  data 
for  implementing  the  method,  to  evaluate  a  prototype  guide  describing 
the  method,  and  to  prepare  a  revised  users  guide  for  the  expert  esti¬ 
mate  method. 
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RESEARCH  APPROACH 


Hi*-'  research  approach  was  similar  to  the  Sauer  and  Askren  (1978a; 
1978b)  study  and  followed  the  recommendations  in  the  prototype  guide 
produced  from  that  study.  An  engineering  description  of  an  operational 
ground  radar  system  was  prepared  to  include  only  information  that 
would  have  been  available  during  the  early  design  phase  of  the  system. 

A  supplement  to  the  engineering  description  (using  early  design  phase 
information  only)  was  also  prepared  to  determine  the  effects  of  addi¬ 
tional  information  on  the  estimates.  Using  only  the  engineering 
description  and  its  supplement,  technicians  with  various  types  of 
ground  radar  maintenance  experience  (one  career  field,  three  Air  Force 
Specialty  Codes  [AFSCs])  estimated  manpower,  maintenance,  and  training 
requirements  for  that  system.  The  technicians'  estimates  were  com¬ 
pared  with  manpower,  maintenance  and  training  data  available  on  the 
operational  system  to  determine  the  validity  of  those  estimates. 

Other  portions  of  the  research  effort  involved  the  collection  of 
critiques  of  the  expert  estimate  prototype  users  guide,  and  collection 
of  data  which  could  be  used  to  estimate  the  cost  of  using  the  expert 
estimate  method. 

Selection  of  Operational  System 

The  AN/TPS-43 (E) ,  a  mobile,  three-d  imensional  ground-based  radar 
system  was  chosen  as  the  test  system.  The  system  had  both  early 
engineering  data  and  operational  data  available.  It  had  been  deployed 
with  Tactical  Air  Command  units  for  approximately  18  months  when  the 
data  collection  effort  took  place  and  the  system  represented  state-of- 
the-art  technology.  It  was  expected  that  an  engineering  description 
of  a  new  state-of-the-art  system  would  appear  as  a  plausible  advanced 
system  to  the  estimators  in  this  study. 

Development  of  Engineering  Description  Package 

The  engineering  description  package  was  developed  following  the 
recommendations  and  examples  provided  in  the  prototype  users  guide 
(Sauer  &  Askren,  1978b).  Engineering  information  was  provided  by 
Tactical  Air  Command  Headquarters.  Electronics  engineers  from  Systems 
Research  Laboratories,  Inc.  (SRL) ,  edited  and  combined  this  informa¬ 
tion  to  form  the  basic  engineering  description  package  and  the  supple¬ 
ment  to  the  basic  package.  The  engineering  description  contained  an 
acronym  glossary,  a  general  system  description,  component  descriptions, 
built-in  test  equipment  descriptions,  illustrations,  and  a  block 
diagram  of  the  system.  The  supplement  contained  additional  engineering 
data  on  system  components. 
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Estimating  Manpower,  Maintenance,  and  Training  Data  Items 


The  data  items  that  the  technicians  were  asked  to  estimate  are 
presented  in  Table  1.  The  sources  of  the  operational  data  used  to 
validate  the  estimates  are  also  presented  in  Table  1.  The  data  items 
include  items  satisfactorily  estimated  in  previous  studies:  person- 
hours,  crew  size,  skill  level,  and  career  field/AFSC.  Other  data 

TABLE  1.  TYPES  OF  DATA  FOR  WHICH  ESTIMATES  WERE 
COLLECTED  AND  SOURCES  OF  CORRESPONDING 
OPERATIONAL  DATA 


Types  of  Data 

Sources  of  Operational  Data 

Person-hours 

AFM  66-1  Data 

Crew  Size 

AN/TPS-43(E)  System  Experts 

Skill  Level 

AN/TPS-43 (E)  System  Experts 

Troubleshooting  Difficulty 

AN/TPS-43 (E)  System  Experts 

Percentage  of  Maintenance 

Task  Occurrences 

AFM  66-1  Data 

Career  Field/AFSC 

AN/TPS-43 (E)  System  Experts 

Training  Times 

AN/TPS-43(E)  System  Experts 

Support  Equipment  Requirements 

AN/TPS-43(E)  Technical  Order 

skill  level,  and  career  field/AFSC.  Other  data  items  which  appeared 
to  have  potential  applications  in  early  system  design  and  planning 
studies  were  also  included:  troubleshooting  difficulty,  percent  of 
maintenance  task  occurrences,  training  times,  and  support  equipment 
requirements.  In  all  cases,  an  item  was  included  only  if  a  satis¬ 
factory  source  of  operational  data  could  be  obtained.  Details  of  each 
data  item  and  its  corresponding  operational  data  set  are  presented  in 
the  following  paragraphs. 

Maintenance  Scenarios.  Maintenance  scenarios  were  developed  to 
help  technicians  make  estimates  of  all  maintenance  and  manpower  items 
except  career  field/AFSC,  training  time  requirements,  and  support 
equipment  requirements.  The  maintenance  scenario  establishes  a  con¬ 
text  or  frame  of  reference  for  making  these  estimates.  It  includes 
four  types  of  information:  (a)  the  general  type  of  maintenance 
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(on-  or  off-equipment),  (b)  the  name  of  the  component  and  its  work 
unit  code  (WUC) ,  (c)  the  type  of  malfunction,  and  (d)  the  specific 
maintenance  action  performed  to  correct  the  malfunction. 

The  estimates  that  technicians  made  for  maintenance  and  manpower 
data  items  were  grouped  by  two  general  types  of  maintenance:  on- 
equipment  maintenance  and  off-equipment  maintenance.  Six  components 
and  their  associated  WUCs  were  selected  for  the  on-equipment  mainte¬ 
nance  estimates.  They  were: 


1. 

RF  (radar  frequency)  receivers 

AAAAD 

2. 

Transmitter 

ABAOO 

3. 

SFg  (sulfur  hexafluoride)  tank 

ABAHO 

4. 

IF  (intermediate  frequency)  receiver 

ABF  00 

5. 

IF  height  receiver 

ABFDO 

6. 

IF  Search/MTI  (moving  target  indicator) 
ABFFO 

Three  malfunctions  and  their  associated  maintenance  actions  were 
identified  for  each  of  the  six  components.  A  total  of  18  maintenance 
scenarios  was  constructed  for  the  estimates  of  on-equipment  mainte¬ 
nance  and  manpower  data.  The  set  of  18  scenarios  is  contained  in 

Table  2. 


It  will  be  noted  that  not  all  scenarios  contained  specific  mal¬ 
functions  or  actions  taken.  The  first  scenario  for  each  component 
(No.  1,  4,  7,  10,  13,  16)  did  not  specify  a  particular  malfunction  or 
maintenance  action.  The  intent  here  was  to  present  to  the  technicians 
a  general  maintenance  scenario  for  each  component.  Estimates  of  task 
times  using  these  scenarios  would  be  similar  to  Mean  Time  to  Repair 
(MTTR)  values  for  the  components.  The  estimates  of  crew  size  and 
skill  levels  and  the  derived  estimates  of  person-hours  for  these 
scenarios  represented  general  work  estimates  of  these  data.  Mainte¬ 
nance  scenarios  for  troubleshooting  tasks  did  not  specify  a  particular 
malfunction. 

Maintenance  scenarios  for  of f-equipment  tasks  are  presented  in 
Table  3.  Each  numbered  column  is  a  separate  scenario.  The  first 
scenario  for  each  component  (No.  1,  3,  5,  7)  was  intended  to  present 
to  the  technicians  a  general  off-equipment  maintenance  scenario.  Task 
time  estimates  for  these  scenarios  would  be  similar  to  the  MTTR  values. 


TABLE  2.  ON-EQUIPMENT  MAINTENANCE  SCENARIOS  I  TO  18 
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Repair  Maintenance 


TABLE  3. 


OFF-EQUIPMENT  MAINTENANCE  SCENARIOS  1  TO  18 


Scenario 

1 

Scenario 

2 

Scenario 

1 

Scenario 

4 

Component 
and  Work 

Unit  Code 

RF  Receiver 
AAAAD 

RF  Receiver 
AAAAD 

Power  Supply 
Focus  Coil 
ABAGO 

Power  Supply 
Focus  Coil 
AGAGO 

How 

Malfunction 

Not 

Specified 

Not 

Specified 

Not 

Specified 

No  Output 

Action 

Taken 

General 

Maintenance 

Any 

NRTS 

Action 

General 

Maintenance 

Shop 

Repair 

Scenario 

5 

Scenario 

6 

Scenario 

7 

Scenario 

8 

Component 
and  Work 

Unit  Code 

IF  Height 
Receiver 

ABFDO 

IF  Height 
Receiver 

ABFDO 

IF  Search/ 

MTI 

Receiver 

ABFFO 

IF  Search/ 
MTI 

Receiver 

ABFfO 

How 

Malfunction 

Not 

Specified 

No 

Output 

Not 

Specified 

Not 

Specified 

Action 

Taken 

General 

Maintenance 

Shop 

Repair 

General 

Maintenance 

Any 

NRTS3 

Action 

aNot  Repairable  This  Station 


Task  time,  crew  size,  and  skill  level  estimates  and  derived  person-hour 
estimates  were  made  for  each  maintenance  scenario  (on-equipment  =  18, 
off-equipment  =8).  In  addition,  troubleshooting  difficulty  estimates 
were  made  for  each  on-equipment  scenario  which  specified  a  trouble¬ 
shooting  task  (n  “  6),  and  percent  of  maintenance  task  occurrences 
estimates  were  made  for  each  of  the  six  components  identified  in  the  on- 
equipment  maintenance  scenarios. 
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person-hours,  however;  they  estimated  task  time  and  crew  size.  A 
person-hour  estimate  was  derived  by  multiplying  the  technician's  task 
time  estimate  by  the  crew  size  estimate.  There  were  two  reasons  for 
using  this  approach.  First,  the  AFM  66-1  (Maintenance  Management 
Policy)  operational  data  which  were  available  were  in  the  form  of 
person-hours.  The  derived  person-hour  estimates,  therefore,  could  be 
directly  compared  to  the  operational  data.  Second,  technicians  do  not 
record  their  maintenance  data  in  terms  of  person-hours.  When  they 
complete  the  maintenance  data  form,  AFTO  Form  349  (Maintenance  Data 
Collection  Record),  they  enter  task  time  and  crew  size,  but  not  person- 
hours.  Technicians  were  therefore  asked  to  estimate  the  same  types  of 
data  they  were  familiar  with  in  their  daily  recordkeeping  activities. 
The  person-hour  operational  data  for  the  on-  and  off-equipment  mainte¬ 
nance  tasks  were  taken  from  AFM  66-1  maintenance  data  for  the  AN/TPS- 
43(E)  radar  system.  On-equipment  person-hour  data  were  for  the  12- 
month  period  beginning  01  October  1977  and  ending  30  September  1978. 

The  off-equipment  person-hour  data  were  for  the  12-month  period 
beginning  01  September  1977  and  ending  31  August  1978. 

Crew  Size  Estimates  and  Operational  Data.  Crew  size  was  defined 
as  the  number  of  technicians  required  to  perform  the  specified  mainte¬ 
nance  tasks.  Technicians  estimated  the  percentage  of  time  that  a 
one-,  two-,  three-,  and  four-member  crew  would  be  required  to  perform 
the  specified  maintenance  task.  The  sum  of  the  percentages  for  a 
given  task  equals  100.  Crew  size  estimates  were  made  for  the  18  on- 
equipment  and  8  off-equipment  maintenance  tasks.  Operational  data  for 
crew  size  were  obtained  from  four  system  experts  who  had  extensive 
maintenance  experience  on  the  AN/TPS~43(E)  (Table  4).  The  system 
experts  provided  the  estimate  of  the  percentage  of  time  a  particular 
crew  size  would  be  required  for  a  given  maintenance  task. 


TABLE  4.  SPECIALTIES,  SKILL  LEVELS,  AND  EXPERIENCE 

OF  THE  FOUR  AN/TSP-43(E)  SYSTEM  PERSONNEL  USED 
AS  EXPERTS  IN  GENERATING  OPERATIONAL  DATA 


Months  of 

AFSC 

N 

Skill  Levels 

Systems  Experience 

Z  1 

Mean 

Range 

303X2 

4 

3  1 

19.5 

18  -  24 

Skill  Level  Estimates  and  Operational  Data.  Skill  level  was 
defined  in  terms  of  the  Air  Force  skill  level  classifications  of  3,  5, 
7,  and  9.  Technicians  estimated  the  skill  level  requirements  in 
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conjunction  with  their  crew  size  estimates.  No  skill  levels  were 
estimated  for  crew  sizes  assigned  a  percentage  score  of  zero.  Oper¬ 
ational  data  for  skill  levels  were  provided  by  the  system  experts 
described  earlier  (Table  4). 

Career  Field/AFSC  Estimates  and  Operational  Data.  Career  field 
was  defined  as  the  first  three  digits  of  the  five-digit  AFSC  assigned 
to  Air  Force  enlisted  personnel.  Technicians  first  were  asked  to 
estimate  career  field  requirements  for  the  new  system.  Technicians 
then  were  asked  to  estimate  either  the  specific  five-digit  AFSC  which 
would  be  assigned  to  maintain  the  system  or  the  actions  which  would  be 
necessary  to  obtain  the  proper  maintenance  personnel  for  the  system. 
System  experts  (Table  4)  provided  the  operational  career  field/AFSC 
requirements  for  the  system. 

Percentage  of  Maintenance  Task  Occurrences  Estimates  and 
Operational  Data.  Percentage  of  maintenance  task  occurrences  was 
defined  as  the  percentage  of  on-equipment  maintenance  tasks  for  a 
particular  component  which  could  be  placed  into  one  of  five  mainte¬ 
nance  categories.  The  five  categories  were:  (a)  Minor  Repair, 

(b)  Install/Remove/Remove  and  Replace,  (c)  Adjust,  (d)  Troubleshoot, 
and  (e)  Other.  Four  categories  represented  the  four  types  of  specific 
maintenance  activities  most  frequently  performed  on  the  ground-based 
radar  system,  and  a  fifth  category  represented  the  remaining  mainte¬ 
nance  activities.  Given  all  the  maintenance  actions  which  could  be 
performed  on  a  component  (100  percent  of  the  maintenance  tasks),  the 
technician  estimated  the  percentage  of  these  tasks  that  would  be 
categorized  into  each  of  the  five  categories.  These  types  of  esti¬ 
mates  provided  an  indication  of  how  the  maintenance  actions  would  be 
distributed  for  a  given  component.  Operational  data  were  the  percen¬ 
tage  of  on-equipment  maintenance  actions  actually  recorded  in  the  five 
categories  for  the  six  system  components.  These  data  were  extracted 
from  AFM  66-1  maintenance  data. 

Troubleshooting  Difficulty  Estimates  and  Operational  Data . 
Troubleshooting  difficulty  was  defined  as  the  degree  of  difficulty 
involved  in  performing  troubleshooting  tasks  for  the  six  system  com¬ 
ponents  for  which  on-equipment  maintenance  tasks  were  specified.  The 
estimates  were  made  on  a  100-mm  scale  ranging  from  0  (Very  Easy)  to 
100  (Very  Difficult).  The  operational  data  were  similar  ratings  made 
by  the  system  experts  (Table  4),  based  on  their  maintenance  experience 
with  the  AN/TPS-43 (E)  radar. 

Training  Time  Estimates  and  Operational  Data.  In  this  study 
maintenance  technicians  estimated  the  weeks  of  technical  training, 
field  training  detachment  training,  and  on-the-job  training  necessary 
to  bring  an  experienced  radar  technician  and  a  new  radar  technician 
(no  previous  radar  experience)  up  to  5-level  maintenance  performance 
on  the  proposed  system.  Previous  attempts  to  collect  training  data 
estimates  involved  very  detailed  questions  on  content  and  times  (Sauer 
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&  Askren,  1978a)  and  resulted  in  inadequate  training  data  estimates. 

A  simpler,  more  direct  approach  was  tried,  commensurate  with  the 
amount  of  engineering  information  available  in  the  early  design  stage 
of  system  development.  fhe  operational  data,  actual  training  times 
for  the  AN/TPS-43(E)  radar,  were  provided  by  the  system  experts 
(Table  4). 

Support  Equipment  Estimates  and  Operational  Data.  Support 
equipment  requirements  were  defined  as  the  types  of  tools  and  test 
equipment  necessary  to  support  the  maintenance  program  for  the  radar 
system.  Technicians  used  an  open-end  response  format  to  record  their 
estimates  of  the  types  of  equipment  necessary.  Operational  data  were 
the  types  of  tools  and  test  equipment  actually  required  to  support 
maintenance  on  the  system.  This  information  was  extracted  from  the 
AN/TPS-43(E)  technical  manuals. 

Collection  of  Additional  Data.  The  technicians  provided  the  fol¬ 
lowing  additional  information:  work  experience,  ratings  of  their 
confidence  in  their  estimates,  the  percent  of  each  estimate  based  on 
their  past  maintenance  experience  and  on  the  engineering  data  pro¬ 
vided,  and  the  effect  the  additional  engineering  data  had  on  their 
estimates. 

Estimator  Personnel 


Personnel  participating  as  expert  estimators  were  radar  mainte¬ 
nance  technicians  from  three  AFSC  groups:  303X1,  air  traffic  control 
radar  technicians;  303X2,  aircraft  control  and  warning  radar  techni¬ 
cians;  and  303X3,  automatic  tracking  radar  technicians.  These  groups 
represented  all  AFSC  groups  within  the  303XX  career  field.  This  is 
the  career  field  one  would  have  selected  in  an  actual  application  of 
the  method  based  on  recommendations  in  the  prototype  guide  (Sauer  & 
Askren,  1978b).  The  number  of  technicians  participating  from  each 
group,  and  the  years  of  maintenance  experience  and  skill  levels 
possessed  by  the  technicians  are  presented  in  Table  5.  None  of  these 
technicians  had  work  experience  on  the  AN/TPS-43 (E) . 
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TABLE  5.  HUMAN  RESOURCES  DATA  ESTIMATOR  CHARACTERISTICS: 

AFSC,  SKILL  LEVELS,  YEARS  OF  AFSC  EXPERIENCE 
[NO  EXPERIENCE  ON  AN/TPS-43 (E) ] 


AFSC 

Group 

Skill 

.  Level 

Years  of  AFSC 
Experience 

N 

3 

_5 

1 

0 

Mean 

Range 

303X1 

16 

1 

4 

10 

1 

8.6 

<1.0-22.3 

303X2 

25 

1 

15 

9 

— 

7.6 

1.4-19.8 

303X3 

29 

7 

12 

10 

— 

4.7 

<1.0-16.0 

Data  Collec 

tion 

Prior 

to  the 

actual 

data 

collection 

effort,  SRL 

employees  with 

previous  Air  Force  radar  maintenance  experience  reviewed  the  engi¬ 
neering  description  package,  the  questionnaire,  and  the  proposed 
procedures  to  be  used  in  the  data  collection  effort  to  identify  any 
unforeseen  data  collection  problems. 

Table  6  contains  a  listing  of  the  units  visited,  their  parent 
commands,  and  the  bases  where  the  units  were  located.  The  data  col¬ 
lection  occurred  in  group  settings  of  from  4  to  15  technicians.  Each 
technician  was  given  a  copy  of  the  basic  engineering  description 
package  and  the  maintenance  manpower  questionnaire.  The  questionnaire 
administrator  briefed  the  technicians  on  the  purpose  of  the  research 
project  prior  to  giving  the  verbal  instructions  for  the  questionnaire. 
Technicians  were  advised  to  read  through  the  entire  basic  engineering 
description  before  attempting  to  make  their  estimates.  The  techni¬ 
cians  were  encouraged  to  refer  to  the  engineering  description  as  often 
as  necessary  during  the  session. 

Upon  completion  of  the  questionnaire,  technicians  were  given  a 
copy  of  the  supplement  to  the  basic  engineering  description  package. 
They  were  instructed  to  read  the  additional  engineering  information 
and  go  back  through  and  reconsider  their  initial  estimates.  If, 
based  upon  the  additional  engineering  information,  they  wished  to 
change  an  estimate,  they  were  instructed  to  circle  the  initial  esti¬ 
mate  and  enter  the  new  estimate  adjacent  to  the  initial  estimate. 

At  this  time,  the  technicians  were  also  given  the  last  page  of  the 
questionnaire  to  collect  their  confidence  ratings  of  their  estimates, 
the  information  sources  used  for  their  estimates,  and  the  effect 
the  supplemental  engineering  data  had  on  their  estimates. 
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TABLE  6.  BASES,  COMMANDS,  UNITS,  AND  TECHNICIAN  ESTIMATORS 
PARTICIPATING  IN  THE  DATA  COLLECTION  EFFORT 


Base 

Command 

Unit 

n 

AFSC 

Port  Austin  AFS 

ADC 

754th  Radar  Sqdn. 

25 

303X2 

Avon  Park  Air  Force 
Range 

TAC 

56th  Combat  Support 
Squadron 

25 

303X3 

Eglin  AFB 

TAC 

75th  Tactical 
Control  Flight 

4 

303X3 

AFCS 

1972  Comm  Sqdn. 

4 

303X1 

Patrick  AFB 

AFCS 

2179  Comm,  and 
Installation  Group 

6 

303X1 

Wright-Patterson  AFB 

AFCS 

2046  Comm,  and 
Installation  Group 

6 

303X1 

Prototype  Users  Guide  Evaluation 

A  total  of  seven  manpower  experts  from  the  Air  Force  Maintenance 
and  Supply  Management  Engineering  Team  and  the  Crew  Equipment  and 
Human  Factors  Division  of  the  ASD  Directorate  of  Equipment  Engineering 
at  Wright-Patterson  AFB,  reviewed  and  critiqued  the  contents  and 
methodology  presented  in  the  prototype  users  guide.  Seven  questions 
were  prepared  and  distributed  with  each  copy  of  the  guide.  The  ques¬ 
tions  addressed  the  clarity  of  the  writing  and  explanations,  the 
adequacy  of  the  information  and  examples  provided,  and  the  feasibility 
of  suggested  applications  for  the  method.  The  last  question  was  open- 
ended  and  solicited  any  additional  comments  and  criticism. 

Cost  Evaluation 


To  evaluate  the  potential  cost  of  collecting  expert  estimates, 
person-hours  were  tracked  for  each  phase  of  the  data  collection 
effort.  Those  person-hours  which  were  required  only  for  execution  of 
the  research  approach,  such  as  selection  of  a  test  system  and  the 
collection  of  operational  data,  were  excluded  from  the  cost  analysis. 
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RESULTS 


Accuracy  Measures 

To  evaluate  the  accuracy  of  the  various  maintenance  and  man¬ 
power  data  estimates,  several  kinds  of  accuracy  measures  were  used. 
For  crew  size,  troubleshooting  difficulty,  and  training  time  esti¬ 
mates,  and  derived  person-hour  estimates,  the  accuracy  index  or 
score  was  calculated  by  dividing  the  estimated  value  by  the  actual 
or  criterion  value  (Estimate  *  Actual  =  Accuracy  Score).  An  accu¬ 
racy  score  of  1.00  indicated  that  the  estimate  was  the  same  as  the 
actual  value.  An  accuracy  score  less  than  1.00  indicated  that  the 
estimated  value  was  less  than  the  actual  value  (underestimate), 
while  an  accuracy  value  greater  than  1.00  indicated  an  estimated 
value  greater  than  the  actual  value  (overestimate).  For  estimates 
of  the  most  likely  crew  size,  skill  levels,  career  field/AFSC,  and 
support  equipment  requirements,  the  percentage  of  estimates  that 
agreed  with  the  actual  or  criterion  value  was  the  indicator  of 
accuracy. 

For  estimates  of  percentage  of  maintenance  task  occurrences, 
the  degree  of  agreement  between  the  list  of  estimated  percentages 
and  the  list  of  actual  percentages  was  examined.  A  chi-square 
analysis  was  used  to  compare  the  estimated  percentages  with  the 
actual  percentages.  No  differences  between  the  estimated  percent¬ 
ages  (observed)  and  the  criteria  percentages  (expected)  would  indi¬ 
cate  that  technicians'  estimates  of  percent  of  maintenance  task 
occurrences  were  similar  to  the  actual  percent  of  occurrences. 

Person-Hour  Values 


Person-hour  values  were  obtained  for  on-  and  off-equipment 
maintenance  tasks.  The  values  were  derived  by  multiplying  each 
technician's  normal  task  time  estimate  by  the  crew  size  that  had 
been  estimated  as  the  most  likely  to  perform  the  particular  task. 

The  mean  person-hour  value  for  a  given  task  was  calculated  for  each 
AFSC  group.  This  mean  person-hour  value  was  compared  to  the  mean 
person-hours  reported  for  that  task  in  the  AFM  66-1  maintenance  data 
base  for  the  AN/TPS-43(E)  radar  system. 

Person-hour  values  and  accuracy  scores  were  calculated  for  18 
on-equipment  maintenance  tasks:  three  maintenance  tasks  for  each  of 
the  six  components/WUCs  (see  Table  2).  For  the  general  maintenance 
tasks  (the  first  maintenance  scenario  for  each  component /WUC) ,  the 
technicians  were  instructed  to  estimate  normal,  minimum,  and  maximum 
task  times  for  general  on-equipment  maintenance  tasks  (except  trouble¬ 
shooting  tasks)  for  the  component.  These  estimates  would  be  similar 
to  those  for  MTTR.  For  the  remaining  maintenance  scenarios,  the 
technicians  estimated  task  times  for  troubleshooting  tasks  and  other 
specific  maintenance  tasks. 
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The  estimating  accuracy  of  the  three  AFSC  groups  was  compared 
for  all  on-equipment  maintenance  task  types.  A  Kruskal-Wallis  one¬ 
way  analysis  of  variance  procedure  indicated  no  significant  dif¬ 
ferences  in  terms  of  accuracy  of  the  estimates  among  the  three  AFSC 
groups  (Kruskal-Wallis  H  =  1.42,  p  <  .50).  Estimating  accuracy, 
however,  did  vary  by  type  of  maintenance  task  (Kruskal-Wallis 
H  =  5.96,  p  <  .02).  The  technicians  produced  the  most  accurate 
estimates  for  the  specific  maintenance  tasks.  General  maintenance 
tasks  were  next  best  in  terms  of  accuracy,  and  troubleshooting  tasks 
had  the  worst  accuracy  record.  A  summary  of  the  mean  person-hour 
values,  the  AFM  66-1  data,  and  the  accuracy  scores  for  the  three 
types  of  on-equipment  tasks  and  the  three  AFSC  groups  is  given  in 
Table  7. 

Off-equipment  maintenance  person-hours  were  calculated  in  the 
same  manner  as  the  on-equipment  maintenance  person-hours.  Techni¬ 
cians  estimated  task  times  for  four  work  unit  codes  and  two  types  of 
off-equipment  tasks:  general  tasks,  which  were  defined  as  all 
possible  shop  actions  for  a  given  work  unit  code;  and  specific 
tasks,  which  were  defined  in  terms  of  a  specific  "how  malfunction" 
code  and  "action  taken"  code  or  as  any  "NRTS”  (Not  Repairable  This 
Station)  action. 

A  Kruskal-Wallis  one-way  analysis  of  variance  indicated  there 
were  no  significant  differences  among  the  three  AFSC  groups  with 
regard  to  the  accuracy  of  off-equipment  maintenance  person-hour 
values  (H  =  3.71,  p  =  .20).  Also  presented  in  Table  7  is  a  summary 
of  mean  person-hour  values,  AFM  66-1  mean  person-hours,  and  accuracy 
scores  for  the  three  AFSC  groups  and  two  types  of  off-equipment 
maintenance  tasks.  In  contrast  to  the  on-equipment  person-hour 
values,  there  were  no  significant  differences  in  the  accuracy  of 
off-equipment  person-hour  values  by  the  types  of  maintenance  tasks 
(Mann-Whitney  U  =  3,  p  =  .35).  Overall,  technicians  tended  to 
underestimate  on-equipment  person-hours  while  off-equipment  person- 
hour  values  for  the  groups  were  both  over  and  under  actual  person- 
hours.  The  accuracy  scores  for  estimates  of  general  shop  actions 
were  very  close  to  1.00  with  perhaps  a  slight  tendency  to  under¬ 
estimate  person-hours.  For  specific  shop  actions,  the  technician 
generally  overestimated  person-hours.  The  overall  accuracy  for  all 
groups  and  both  types  of  maintenance  was  .80.  For  comparison  pur¬ 
poses,  the  accuracy  values  reported  previously  were  .66  (Sauer  & 
Askren,  1978a)  and  .70  (Whalen  &  Askren,  1974). 

Crew  Size  Estimates 

Technicians  estimated  crew  size  requirements  in  terms  of  the 
percentage  of  time  a  one-,  two-,  three-,  and  four-member  crew  was 
required  to  perform  the  specified  maintenance  task.  For  example,  a 
technician  who  estimated  a  value  of  10  for  a  one-member  crew  and  a 


TABLE  7.  AFM  66-1  PERSON-HOUR  DATA,  MEAN  PERSON-HOUR  VALUES 
AND  ACCURACY  SCORES  FOR  ON-  AND  OFF-EQUIPMENT 
MAINTENANCE  TASKS  FOR  THREE  AFSC  GROUPS 


'Accuracy  Score  =  n  Person-hour  Values  for  Task  Type 

Mean  AFM  66-1  Person-hours  for  Task  Type 


; 


value  of  90  for  a  two-member  crew  was  stating  that,  90  percent  of 
the  time,  two  persons  would  be  required  to  perform  the  task,  and 
10  percent  of  the  time,  one  member  would  be  required  to  perform  the 
task.  In  this  example,  three-  and  four-member  crews  were  considered 
unnecessary  for  performance  of  the  task  and  were  therefore  assigned 
values  of  0. 

The  first  analysis  conducted  on  the  crew  size  estimates  was  to 
determine  how  well  the  AFSC  groups  estimated  the  crew  size  most 
likely  to  perform  the  specified  tasks.  The  accuracy  measure  used 
here  was  the  percentage  of  times  the  technicians  correctly  estimated 
the  most  likely  crew  size  for  both  on-  and  off-equipment  maintenance 
tasks.  These  percentages  (Table  8)  ranged  from  41.6  percent  for  the 
303X3  technicians,  45  percent  for  the  303X1  technicians,  to  70  percent 
for  the  303X2  technicians. 

TABLE  8.  PERCENT  OF  MAINTENANCE  TASKS  WHERE  ESTIMATORS 
CORRECTLY  INDICATED  THE  CREW  SIZE  MOST  LIKELY 
TO  PERFORM  THE  TASK 


AFSC  Groups 


Maintenance  Tasks 

303X1 

303X2 

303X3 

On-Equipment 

General  Tasks 

33% 

100% 

100% 

Troubleshoot  Tasks 

50% 

50% 

50% 

Specific  Tasks 

67% 

50% 

33% 

Of  f-Equipment 

General  Tasks 

25% 

75% 

25% 

Specific  Tasks 

50% 

75% 

0% 

Overall 

45% 

70% 

41.6; 

Although  no  statistically  significant  differences  were  found 
among  the  groups  (Kruskal-Wallis  H  =*  3.67,  p  <  .20),  the  303X2 
technicians  appeared  to  make  more  accuiate  estimates  of  crew  size 
than  did  the  other  two  groups. 

An  analysis  of  the  incorrect  estimates  indicated  that  the 
magnitude  of  the  errors  never  exceeded  plus  or  minus  one  crew 
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member.  The  tendency  was  for  technicians  to  estimate  larger  rather 
than  smaller  crew  sizes.  They  overestimated  crew  size  by  one  in  28 
of  the  36  tasks  (78  percent)  for  which  incorrect  crew  size  estimates 
were  made.  For  the  remaining  eight  tasks  (22  percent)  they  under¬ 
estimated  crew  size  by  one. 

The  mean  accuracy  scores  by  AFSC  group  for  estimating  the 
percentage  of  time  crew  sizes  of  one  and  two  are  required  are  pre¬ 
sented  in  Table  9.  The  accuracy  score  was  calculated  by  dividing 
the  estimated  percent  of  time  a  one-  and  two-member  crew  would  be 
required  by  the  actual  percent  of  time  a  one-  and  two-member  crew 
would  be  required. 


TABLE  9.  MEAN  ACCURACY  SCORES  FOR  ONE-  AND  TWO-MEMBER 
CREW  SIZE  ESTIMATES 


Mean  Accuracy 

Scores3 

by  AFSC  Groups 

Crew 

Maintenance  Tasks 

Size 

303X1 

303X2 

303X3 

On- Equipment 

General  Tasks 

1 

1.78 

1.28 

.05 

2 

.71 

.94 

1.02 

Troubleshooting 

1 

1.91 

1.80 

.06 

Tasks 

2 

.91 

1.05 

1.11 

Specific  Tasks 

1 

1.93 

1.29 

.03 

2 

1.35 

2.17 

3.22 

Of  f-Equ i pment 

General  Tasks 

1 

1.85 

1.41 

.19 

2 

.99 

1.07 

1.35 

Specific  Tasks 

I 

.60 

.50 

.  10 

2 

5.01 

5.62 

7.48 

Overall 

1 

1.61 

1.26 

.09 

Overall 

2 

1.83 

2.17 

2.84 

aAccuracy  Score  =* 

Estimated  Percent  of  Times  a  Crew  Size 

is  Required 

Actual 

Percent 

of  Times  a  Crew 

Size  is 

Requ i red 

The  accuracy 

with  which  the 

AFSC  groups  estimated  tl 

le  percent 

of  time  a  one-member  crew 

would 

be  required  to  pe 

rform  a 

given  task 

22 


differed  significantly  among  the  groups  (Kruskal-Wallis  H  =  8.42, 
p  <  .02).  The  303X1  and  303X2  technicians  had  overall  accuracy 
scores  of  1.61  and  1.26,  respectively,  indicating  they  overestimated 
the  percentage  of  time  a  one-member  crew  would  be  required.  The 
overall  accuracy  scores  for  the  303X2  group,  however,  demonstrated 
an  acceptable  degree  of  estimating  accuracy.  The  303X3  technicians 
had  an  overall  accuracy  score  of  .09,  indicating  they  greatly  under¬ 
estimated  the  percentage  of  times  a  one-member  crew  would  be  required. 
No  such  accuracy  differences  were  found  among  the  AFSC  groups  for 
estimates  of  two-member  crew  requirements  (Kruskal-Wallis  H  =  .38, 
p  <  .90).  The  technicians  generally  overestimated  the  percent  of 
times  a  two-member  crew  would  be  required  for  a  given  task. 

Crew  size  estimates  for  one-  and  two-member  crews  were  also 
compared  over  the  various  types  of  maintenance  tasks.  No  differ¬ 
ences  were  found  in  terms  of  accuracy  of  crew  size  estimates  over 
the  three  types  of  on-equipment  tasks:  general,  troubleshooting, 
and  specific  (Kruskal-Wallis  H  =  3.26,  p  <  .20).  However,  for  the 
off-equipment  tasks,  crew  size  estimates  were  more  accurate  for  the 
general  tasks  than  for  the  specific  tasks  (Mann-Whitney  U  =  3, 
p  =  .008).  No  analyses  were  conducted  on  the  three-  and  four-member 
crew  size  estimates  because  of  the  relatively  few  estimates  made  for 
these  crew  sizes. 

It  appears  that  for  both  types  of  crew  size  estimates,  the  most 
likely  crew  size  and  the  percentage  of  times  crews  of  one  and  two 
members  are  required,  the  303X2  technicians  performed  somewhat 
better  than  either  of  the  other  AFSC  groups.  Where  errors  occurred, 
the  tendency  was  for  technicians  to  overestimate.  For  most  likely 
crew  size  estimates,  the  majority  of  the  errors  were  overestimates 
of  crew  sizes  by  one  crew  member.  Technicians  also  overestimated 
the  percentage  of  times  one-  and  two-member  crews  were  required. 

The  magnitude  of  these  overestimates  appeared  to  be  slightly  less 
for  a  one-member  crew  than  for  a  two-member  crew. 

Skill  Level  Estimates 


Skill  level  estimates  (3,  5,  7,  or  9)  were  made  to  correspond 
to  each  crew  size  estimate.  The  percentage  of  correct  skill  level 
estimates,  when  compared  to  the  operational  data,  was  the  measure 
used  in  these  analyses.  For  crew  sizes  greater  than  one,  all  skill 
levels  must  agree  with  the  criterion  skill  levels  for  the  particular 
maintenance  scenario  to  be  considered  a  correct  response.  No  par¬ 
tial  credit  was  allowed  if  only  one  of  two  or  more  skill  level 
estimates  was  correct.  For  example,  the  operational  data  may  indi¬ 
cate  that  for  a  crew  of  two,  a  5-level  and  a  7-level  technician  are 
required.  The  only  correct  estimate  in  this  case  is  a  5-  and  7- 
level  technician  team.  A  5-  and  5-level  team,  a  5-  and  3-level 
team,  or  any  other  combination  other  than  the  5-  and  7-level  team 
was  considered  incorrect  in  these  analyses.  This  measure  of  correct 
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skill  level  estimates  could  be  considered  a  conservative  measure  for 
crew  sizes  greater  than  one.  The  percentage  of  correct  skill  level 
estimates  for  the  AFSC  groups  over  the  various  types  of  maintenance 
tasks  is  displayed  in  Table  10.  Analyses  for  three-  and  four-member 
crews  were  not  included  due  to  the  limited  data  for  these  crew 
sizes. 


TABLE  10.  PERCENT  OF  CORRECT  SKILL  LEVEL  ESTIMATES 
FOR  ONE-  AND  TWO-MEMBER  CREWS 


Percent  of  Correct  Skill  Level 
Estimates  for  AFSC  Groups 


Maintenance  Tasks 

Crew 

Size 

303X1 

303X2 

303X3 

On-Equipment 

General  Tasks 

1 

65.5 

57.2 

48.8 

2 

33.3 

17.0 

3.0 

Troubleshooting 

1 

66.8 

57.5 

33.3 

Tasks 

2 

35.8 

16.3 

5.2 

Specific  Tasks 

1 

83.3 

71.7 

39.0 

2 

61.5 

55.5 

58.8 

Of  f-Equipment 

General  Tasks 

1 

76.7 

65.7 

35.7 

2 

53.7 

50.0 

35.5 

Specific  Tasks 

1 

98.0 

80.0 

53.3 

2 

73.3 

69.0 

62.3 

Mean  Percentage  of 

1 

78.1 

66.4 

42.0 

Correct  Estimates 

2 

51.5 

41.6 

33.0 

Considering  overall  performance  for  estimating  skill  level, 
significant  differences  were  found  among  the  three  AFSC  groups 
(Kruskal-Wallis  H  **  7.76,  p  <  .05).  The  303X1  group  had  the  highest 
percentage  of  correct  estimates  and  the  303X3  group  had  the  lowest 
percentage  of  correct  estimates.  This  finding  was  heavily  influenced 
by  the  groups'  performance  differences  on  skill  level  estimates  for 
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one-member  crews  (Kruskal-Wallis  H  =  10.22,  p  <  .0097).  There  were 
no  significant  differences  among  the  groups  for  skill  level  esti¬ 
mates  for  two-member  crews  (Kruskal-Wallis  H  =  1.14,  p  <  .70). 

In  comparing  skill  level  estimates  (crew  sizes  of  one  and  two) 
for  the  different  on-equipment  maintenance  tasks,  the  data  suggested 
that  the  skill  level  estimates  were  more  accurate  for  specific  on- 
equipment  tasks  than  for  the  general  and  troubleshooting  tasks 
(Kruskal-Wallis  H  =  4.64,  p  <  .10).  For  off-equipment  tasks, 
however,  skill  level  estimates  were  clearly  superior  for  the  spe¬ 
cific  tasks  (Mann-Whitney  U  =  7,  p  =  .047). 

As  indicated  above,  the  percentage  of  correct  skill  level 
estimates  is  a  conservative  measure  for  crew  sizes  greater  than  one. 
For  77  out  of  78  skill  level  estimates  for  crew  sizes  of  two,  at 
least  one  of  the  skill  levels  estimated  for  the  two-member  crew  was 
correct.  To  determine  the  nature  of  the  errors  in  the  skill  level 
estimates,  an  analysis  of  the  incorrect  skill  level  estimates  for 
crews  of  one  and  two  was  conducted.  It  was  found  that  technicians 
generally  underestimated  skill  level  requirements.  Overall,  28.4  per¬ 
cent  of  the  skill  level  estimates  for  crews  of  one  and  two  indicated 
that  a  numerically  lower  skill  level  or  combination  of  skill  levels 
was  estimated  for  the  task.  Overestimates  of  skill  level  require¬ 
ments  occurred  for  19.6  percent  of  the  estimates.  Overall  for  crew 
sizes  of  one  and  two,  52.0  percent  of  the  skill  level  estimates  were 
correct . 

Troubleshooting  Difficulty  Estimates 

The  troubleshooting  difficulty  estimates  were  made  on  a  100-mm 
scale  with  verbal  anchors  of  Very  Easy  (0  mm).  Average  Difficulty 
(50  mm),  and  Very  Difficult  (100  mm).  Technicians  estimated  the 
degree  of  troubleshooting  difficulty  for  six  system  components. 
Troubleshooting  tasks  represented  on-equipment  maintenance.  The 
AN/TPS-43(E)  system  experts  used  the  same  scale  to  generate  the 
criteria  troubleshooting  difficulty  ratings.  The  criteria  trouble¬ 
shooting  difficulty  ratings,  group  estimates  of  troubleshooting 
difficulty,  and  the  accuracy  scores  for  the  six  work  unit  codes 
representing  the  six  system  components  are  depicted  in  Table  11.  No 
differences  were  found  among  the  AFSC  groups  in  terms  of  accuracy  of 
troubleshooting  difficulty  ratings.  The  accuracy  score  for  the 
groups  was  1.28,  indicating  a  tendency  to  overestimate  the  difficulty 
of  these  troubleshooting  tasks.  An  analysis  of  the  accuracy  scores 
by  system  component,  however,  indicated  significant  differences 
in  accuracy  occurred  depending  on  the  component  (Kruskal-Wallis  H  = 
15.15,  p  <  .02).  An  inspection  of  the  accuracy  scores  indi¬ 
cates  that  this  difference  may  be  due  to  the  fact  that  technicians 
greatly  overestimated  the  troubleshooting  difficulty  for  the  RF 
receivers  (Work  Unit  Code  AAAAD) .  Considering  the  accuracy  scores 
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TABLE  11.  TROUBLESHOOTING  DIFFICULTY  RATINGS  AND  ACCURACY 
SCORES  FOR  SIX  ON-EQUIPMENT  WORK  UNIT  CODES 


Troubleshooting  Difficulty  Ratings  (R)  and 
Accuracy  Scores  (A)  for  AFSC  Groups _ 


303X1  303X2  303X3 

Criterion 


wuc 

Rating 

R 

A 

R 

A 

R 

A 

AAAAD 

15 

46 

3.07 

50 

3.33 

51 

3.40 

ABAOO 

63 

53 

oo 

49 

.78 

55 

.87 

ABAHO 

93 

57 

.61 

42 

.45 

52 

.56 

ABFOO 

53 

52 

.98 

48 

.91 

59 

1.11 

ABFDO 

53 

59 

1.11 

50 

.94 

56 

1.06 

ABFFO 

53 

51 

.96 

50 

.94 

55 

1.04 

a 


Accuracy  Score  = 


Estimated  Difficulty  Rating 
Criterion  Difficulty  Rating 


for  all  work  unit  codes  except  AAAAD,  the  overall  mean  accuracy 
score  drops  to  .88,  indicating  that  technicians  tended  to  under¬ 
estimate  the  troubleshooting  difficulty  for  these  components.  An 
inspection  of  Table  11  suggests  that  this  accuracy  score  (excluding 
the  AAAAD  data)  may  be  a  better  indicator  of  the  technicians’  ability 
to  estimate  troubleshooting  difficulty. 

Percent  of  Maintenance  Task  Occurrence  Estimates 


Technicians  were  asked  to  estimate  the  percentage  of  time 
certain  types  of  maintenance  tasks  would  be  performed  on  a  given 
equipment  component  over  a  specific  period  of  time.  Estimates  were 
made  in  terms  of  the  percentage  of  maintenance  tasks  performed  on  a 
component  which  could  be  categorized  into  five  task  types:  minor 
repair,  install/remove/remove  and  replace,  adjust,  troubleshoot,  and 
other  maintenance.  The  estimates  for  the  five  task  categories  for 
any  equipment  component  total  100  percent.  These  estimates  were 
made  for  six  system  components  (six  work  unit  codes).  The  estimates 
for  a  given  component  were  compared  to  the  frequency  of  occurrence 
of  the  tasks  as  determined  from  AFM  66-1  maintenance  data.  A  chi- 
square  analysis  was  performed  to  compare  the  estimate  percentages 
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with  the  AFM  66-1  maintenance  percentages  (frequency  of  occurrence 
per  100  maintenance  actions).  If  no  statistical  differences  were 
found  between  the  expected  frequencies  (AFM  66-1  data)  and  the 
observed  frequencies  (estimates),  one  could  conclude  that  the  tech¬ 
nicians  were  doing  a  fairly  good  job  of  estimating  the  percentage  of 
maintenance  task  occurrences.  However,  the  values  of  chi-square  for 
the  AFSC  groups  and  six  system  components  as  presented  in  Table  12 
ranged  from  x2  =  8.28,  p  <  .10,  to  x2  =  399.94,  p  <  .001.  These 
values  indicate  that  the  technicians'  estimates  of  the  percentages 
of  maintenance  task  occurrences  differ  significantly  from  the  per¬ 
centages  reported  in  the  AFM  66-1  data.  Technicians  did  not  demon¬ 
strate  that  they  could  accurately  estimate  the  distribution  of 
maintenance  tasks  for  a  given  component. 

Career  Field/AFSC  Estimates 


Technicians  first  estimated  the  career  field  and  then  the 
specific  AFSC  which  would  have  responsibility  for  the  maintenance  of 
the  radar  system.  Presented  in  Table  13  is  the  percentage  of  cor¬ 
rect  estimates  for  both  career  field  and  AFSC  for  the  three  esti¬ 
mator  groups.  The  results  indicate  that  technicians  can  estimate 
career  field  requirements  very  well.  Although  the  AFSC  estimates 
exhibit  more  variability  than  the  career  field  estimates,  the 
majority  of  each  estimator  group  correctly  identified  the  required 
AFSC  for  the  radar  system. 
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TABLE  12.  CHI-SQUARE  VALUES  RESULTING  FROM 

ANALYSES  OF  ESTIMATED  AND  ACTUAL  PERCENT 
OF  MAINTENANCE  TASK  OCCURRENCES  FOR 
SIX  SYSTEM  COMPONENTS  BY  AFSC  GROUPS 


X2  Values  by 
Estimator  Groups 


Component 

303X1 

303X2 

303X3 

RF  Receiver 

AAAAD 

399.94 

281.73 

287.10 

Transmitter 

ABAOO 

47.46 

93.93 

106.76 

SF6  Tank 

18.68 

13.80 

16.82 

ABAHO 

IF  Receiver  Assy. 

ABFOO 

154.69 

195.72 

195.48 

IF  Height  Receiver 

ABFDO 

24.42 

11.44 

12.66 

IF  Search/MTI 

Receiver 

ABFFO 

12.95 

8.28 

20.97 

Note:  Critical  Values  of 

X2  with  4  df : 

X2  7.78,  p  < 

.10 

X2  9. 49,  p  <  .05 
X2  13.78,  p  <  .01 
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TABLE  13.  CAREER  FIELD  AND  AFSC  ESTIMATES  BY  ESTIMATOR  GROUPS 


Estimator  Groups 


Career  Field  Et  imates 

303X1 

303X2 

303X3 

303XX3 

88% 

96% 

96% 

Other 

12% 

4% 

4% 

100% 

100% 

100% 

AFSC  Estimates 

303X1 

31% 

8% 

303X2b 

38% 

92% 

64% 

303X3 

24% 

Other 

31% 

8% 

4% 

100% 

100% 

100% 

Actual  Career  Field. 
b Actual  AFSC. 


Time  To  Train  Estimates 


Technicians  were  asked  to  estimate  the  number  of  weeks  of 
technical  training,  field  training  detachment  (FTD)  training,  and 
on-the-job  training  which  would  be  necessary  for  a  technician  to 
achieve  5-level  maintenance  proficiency  on  the  proposed  system.  Two 
types  of  technicians  were  specified:  a  technician  with  radar  mainte¬ 
nance  experience  transitioning  to  the  new  system  and  a  prospective 
radar  technician  recently  graduated  from  basic  training.  The  esti¬ 
mated  training  times  (in  weeks)  were  compared  to  the  training  times 
experienced  or  observed  by  the  AN/TPS-43(E)  experts.  The  criteria 
training  times,  estimated  training  times,  and  accuracy  scores  for 
the  three  AFSC  groups  are  given  in  Table  14.  No  differences  were 
found  among  the  AFSC  groups  with  respect  to  the  accuracy  of  the 
training  time  estimates  (Kruskal-Wallis  H  ■  .67,  p  <  .80).  However, 
significant  accuracy  differences  were  found  depending  on  the  type  of 
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TABLE  14.  TRAINING  TIME  ESTIMATES  AND  ACCURACY  SCORES  FOR  AFSC  GROUPS 
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training  for  which  estimates  were  made  (Kruskal-Wallis  H  =  6.86, 
p  <  .05).  The  field  training  detachment  estimates  appeared  to  be 
responsible  for  this  difference.  Technicians  considerably  over¬ 
estimated  the  training  time  required  for  this  FTD  training.  The 
technicians'  estimates  of  training  times  for  the  other  types  of 
training  came  much  closer  to  the  actual  training  times.  Although 
technicians  generally  overestimated  (overall  accuracy  value  =  1.27) 
training  time,  the  estimates  would  appear  to  be  of  value  in  the 
early  design  stages. 

Support  Equipment  Requirements  Estimates 

Technicians  were  asked  to  estimate  the  support  equipment 
requirements  for  the  new  system,  particularly  the  test  equipment  and 
special  tools  used  to  support  the  maintenance  of  the  system.  Tech¬ 
nicians  were  to  list  as  many  types  of  equipment  and  tools  they  felt 
would  be  necessary  to  support  the  maintenance  on  the  system.  The 
technicians'  responses  were  compared  to  a  list  of  28  different 
generic  (as  opposed  to  specific  models  or  manufacturers)  types  of 
test  and  support  equipment  associated  with  the  AN/TPS-43(E)  radar 
system  to  gauge  how  well  they  could  predict  support  equipment 
requirements.  The  responses  to  this  item  were  not  encouraging  in 
terms  of  either  quality  or  quantity  of  the  responses.  Of  the  28 
types  of  test  and  support  equipment  associated  with  the  system,  the 
AFSC  groups  identified  only  from  14  to  17  of  these  equipment  types. 
Further,  the  group  average  response  rate  for  any  one  of  the  identi¬ 
fied  equipment  types  ranged  from  6.3  percent  to  21.5  percent,  indi¬ 
cating  a  low  response  level  for  this  item.  Although  the  technicians 
were  asked  to  identify  the  different  generic  types  of  equipment, 
fairly  large  percentages  (80  percent  and  52  percent)  of  two  AFSC 
groups  stated  that  "general  support"  equipment  would  be  required. 

It  was  not  possible  to  interpret  these  responses  accurately  since 
each  AFSC  group,  and  probably  each  individual,  had  somewhat  dif¬ 
fering  definitions  of  what  constitutes  "general  support”  equipment. 

Feedback  from  Technician  Estimators 


Effect  of  Additional  Engineering  Detail  on  Maintenance  Manpower 
Estimates.  Technicians  used  the  basic  engineering  data  package  to 
make  their  initial  maintenance  manpower  estimates.  They  were  then 
given  a  supplement  to  the  basic  engineering  data  package  and  instruc¬ 
ted  to  review  all  their  initial  estimates,  considering  the  supple¬ 
mental  information  as  well  as  the  engineering  information  in  the 
basic  package.  If  they  wished  to  change  any  estimate  based  on  the 
supplemental  information,  they  could  do  so  by  circling  their  origi¬ 
nal  estimate  and  recording  their  new  estimate  next  to  the  original. 
Technicians  were  also  asked  what  effect  the  additional  information 
had  on  their  maintenance  manpower  estimates. 
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An  examination  of  the  questionnaires  and  the  technicians' 
statements  as  to  the  effect  of  the  additional  information  revealed 
that  only  a  very  few  of  the  estimates  were  changed  as  a  result  of 
the  additional  information.  Overall,  only  .73  percent  of  the  esti¬ 
mates  made  by  an  AFSC  group  were  changed  as  a  result  of  the  supple¬ 
mental  engineering  data.  For  22.73  percent  of  the  estimates, 
technicians  reported  that  the  additional  information  confirmed  their 
original  estimates.  For  the  majority  of  estimates  (76.62  percent), 
however,  technicians  reported  that  the  additional  engineering  data 
neither  confirmed  nor  changed  their  original  estimates.  The  addi¬ 
tional  engineering  data  had  virtually  no  effect  on  the  original 
maintenance  manpower  estimates. 

Technicians'  Confidence  in  the  Estimates.  Technicians  were 
also  asked  to  provide  a  confidence  rating  of  their  estimates  using  a 
three-point  scale.  The  scale  anchors  were  Little  Confidence  (1), 
Confident  (2),  and  Very  Confident  (3).  Overall,  the  confidence 
scores  for  the  three  career  field  groups  were  1.69,  1.86,  1.89 
(Table  15),  indicating  that  the  groups  were  slightly  less  than 
confident  about  their  estimates.  Looking  at  the  confidence  ratings 
for  specific  types  of  estimated  data  (rows  of  Table  15),  mean  con¬ 
fidence  ratings  ranged  from  1.60  (maintenance  task  time  estimates 
for  specific  off-equipment  tasks)  to  2.14  (career  field/AFSC  estimates). 

Information  Sources  Used  to  Make  Estimates.  To  gain  an  insight 
into  what  kinds  of  information  technicians  used  to  make  their  esti¬ 
mates,  they  were  asked  to  indicate  what  percent  of  each  estimate  was 
based  on  their  maintenance  experience  and  background  and  what  per¬ 
cent  of  each  estimate  was  based  on  the  engineering  information 
provided.  The  resulting  percentages  for  the  various  types  of  data 
estimated  and  the  three  AFSC  groups  are  displayed  in  Table  16.  For 
all  types  of  estimates,  technicians  depended  more  heavily  on  their 
maintenance  background  and  experience  than  on  the  engineering  data 
provided  to  make  the  estimates.  The  data  indicate  that  for  all 
three  AFSC  groups,  67.4  percent  of  a  given  estimate  was  based  on  the 
technicians'  previous  maintenance  experience,  while  32.6  percent  of 
a  given  estimate  was  based  on  information  the  technician  obtained 
from  the  engineering  data  package. 

Evaluation  of  Prototype  Users  Guide 

The  prototype  users  guide  was  reviewed  and  evaluated  by  seven 
manpower  experts  from  the  Air  Force  Maintenance  and  Supply  Manage¬ 
ment  Engineering  Team  and  the  Crew  Equipment  and  Human  Factors 
Division  of  the  ASD  Directorate  of  Equipment  Engineering  at  Wright- 
Patterson  AFB.  The  level  of  work  experience  for  the  individuals 
ranged  from  3  to  20  years,  with  a  mean  level  of  8.7  years. 

The  evaluation  was  designed  to  generate  comments  and  criticisms 
in  two  general  areas:  the  clarity  of  the  guide's  instructions  and 
examples  and  the  suggested  and  potential  utility  of  the  expert 
estimate  method. 
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TABLE  15.  TECHNICIANS’  RATINGS  OF  CONFIDENCE  IN  THEIR  MAINTENANCE  MANPOWER  ESTIMATES 


Note:  Confidence  Rating  Scores:  1  *  Little  Confidence,  2  =  Confident,  3  =  Very  Confident. 


TABLE  16.  PERCENT  OF  TECHNICIANS'  ESTIMATES  BASED  ON  THEIR  MAINTENANCE  EXPERIENCE 
AND  ON  THE  ENGINEERING  DATA  PROVIDED  WITH  THE  QUESTIONNAIRE 
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In  terms  of  clarity,  the  areas  of  the  guide  causing  the  most 
confusion  were  those  describing  the  adjustments  or  corrections  which 
could  be  applied  to  the  estimated  data.  Specifically,  the  guide  did 
not  adequately  describe  how  the  correction  factors  were  determined 
from  the  basic  studies  upon  which  the  recommendations  in  the  guide 
were  based. 

Other  comments  regarding  clarity  were  directed  toward  the 
engineering  description  package  and  the  questionnaire.  It  was 
recommended  that  the  engineering  description  package  contain  less 
information  on  the  fundamental  theory  of  system  operations  (e.g., 
Doppler  theory)  and  more  information  on  the  maintenance  environment, 
the  physical  location  of  the  components,  and  steps  necessary  to 
access  the  components.  In  terms  of  the  questionnaire  development, 
the  evaluators  cited  a  lack  of  guidance  for  preparing  the  mainte¬ 
nance  scenarios  which  provide  a  frame  of  reference  for  the  task 
time,  person-hour,  crew  size,  and  skill  level  estimates.  Speci¬ 
fically,  no  guidance  was  offered  which  indicated  how  the  equipment 
malfunctions  and  how  the  maintenance  actions  were  selected  for  these 
scenarios.  In  general,  however,  the  evaluators  felt  that  the  guide 
contained  sufficient  information  to  apply  the  expert  estimate  method 
in  the  collection  of  system  human  resources  data. 

With  regard  to  the  potential  application  of  the  expert  estimate 
method,  the  evaluators  agreed  that  the  most  appropriate  application 
would  be  in  the  early  design  stages  of  a  system,  specifically 
Milestone  0  and  Milestone  1  in  the  Defense  Systems  Acquisition 
Review  Process.  The  major  limitation  identified,  however,  was  that 
while  the  method  can  be  used  to  estimate  maintenance  manpower  require¬ 
ments  for  individual  line  replaceable  units,  it  does  not  provide  a 
means  of  combining  these  LRU  estimates  into  a  systems-level  esti¬ 
mate.  To  accomplish  this  task,  the  evaluators  indicated  that  con¬ 
sideration  must  also  be  given  to  the  expected  operations  scenario, 
maintenance  demand,  failure  rates,  and  system  activity. 

The  comments  from  these  evaluators  were  taken  into  consider¬ 
ation  in  revising  the  prototype  users  guide.  Where  possible,  changes 
in  the  guide  were  made  to  reflect  these  comments.  The  comments  also 
aided  in  identification  of  future  research  direction. 

Costs  Involved  in  the  Expert  Estimate  Method 

The  person-hours  spent  only  on  the  expert  estimate  activities 
of  the  study  were  recorded  for  potential  use  in  predicting  the 
person-hour  costs  involved  in  similar  applications  of  the  expert 
estimate  method.  Where  activities  were  repeated  several  times 
during  the  study,  the  longest  time  recorded  for  these  activities, 
rather  than  an  average  time,  was  used  as  the  basis  for  predicting 
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the  person-hours.  This  approach  was  used  to  avoid  an  unrealis¬ 
tically  optimistic  prediction  of  the  person-hour  requirements. 

Items  for  which  costs  could  not  be  stated  in  terms  of  person-hours 
were  identified  and  listed  so  that  they  would  be  included  in  a  cost 
analysis  of  a  specific  application  of  the  expert  estimate  method. 

The  cost  estimates  were  divided  into  five  areas:  Engineering 
Description  Package  Development,  Questionnaire  Development,  Expert 
Estimate  Collection,  Data  Reduction  and  Analysis,  and  Report  Pre¬ 
paration.  Table  17  contains  these  cost  estimates.  No  other  cost 
data  were  available  to  indicate  how  these  predicted  costs  would  vary 
for  a  larger-scaled  application  of  the  method. 


TABLE  17.  EXPERT  ESTIMATE  METHOD  COST  DATA 


Activity  Person-hours 


ENGINEERING  DESCRIPTION  PACKAGE  DEVELOPMENT 

Engineering  Data  Collection: 

Data  Collector  80 

Proposed  System  Engineers/Designers  AO 

Engineering  Data  Edit  AO 

Test  Reading  by  Maintenance  Technicians  20 

Drafting  (Artistic  Support)  60 

Technical  Typing/Editing/Reproduction  30 

Subtotal:  270 

QUESTIONNAIRE  DEVELOPMENT 

Selection  of  Items  to  be  Estimated  A0 

Draft  Sections  Following  Format  Suggestions 

in  Users  Guide  AO 

Technical  Typing/Editing/Reproduction  25 

Subtotal:  105 

COLLECTING  EXPERT  ESTIMATES 

Select  and  Locate  Expert  Estimators  20 

Data  Collection  Visit  Scheduling  and 

Coordination  20 

Visit  Request  Preparation  10 

On-Site  Data  Collection  (25  Expert  Estimators) 

Data  Collector  Time  20 

(Assumptions — A  hours  per  session, 

five  estimators  per  session  to  accommodate 

daily  work  scheduling) 
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TABLE  17.  EXPERT  ESTIMATE  METHOD  COST  DATA  (concluded) 


Activity 

Expert  Estimator  Time 

(Assumptions — 1  minute  per  page  of 
engineering  data  (25  pages),  1  minute  per 
estimate  (95  estimates)  plus  20-30  minutes 
introductory  remarks  and  instructions) 

Supervisor  Time  for  Scheduling  Sessions, 

In-  and  Out-Briefings,  other  administrative 


tasks  5 

Subtotal:  137.5 

DATA  REDUCTION  AND  ANALYSIS 

Reduction  of  Raw  Data  10 

Data  Analysis  resulting  in  estimates  30 

Subtotal:  40 


Person-hours 

62.5 


(Assumptions — 25  expert  estimators, 

95  estimates  per  individual) 

REPORT  PREPARATION 

60 

20 

25 
l05 

Also  consider  travel  time,  transportation, 
and  per  diem  if  a  headquarters  out- 
briefing  is  required 


Report  and  Briefing  Preparation 
Graphics  Support 

Technical  Typing/Edi ting /Reproduct ion 

Subtotal : 


Overall  Total:  657.5 


DISCUSSION 


The  main  objectives  of  this  study  were  to  exercise  the  expert 
estimate  method,  to  evaluate  the  recommendations  contained  in  the 
prototype  users  guide,  to  collect  additional  evidence  of  the  accu¬ 
racy  of  the  estimated  data,  to  collect  cost  data  associated  with  the 
method,  and  to  evaluate  and  revise  the  prototype  users  guide  (Sauer 
&  Askren,  1978b).  The  utility  of  the  expert  estimate  method  would 
be  determined  in  large  part  by  the  quality  of  the  estimated  mainte¬ 
nance  manpower  data  for  the  ground-based  system.  The  results  of 
this  study  indicated  that  the  expert  estimate  method  is  a  poten¬ 
tially  useful  means  of  generating  maintenance  manpower  data  for  new 
systems . 

Prototype  Users  Guide  Recommendations 

The  recommendations  in  the  users’  guide  (Sauer  &  Askren,  1978b) 
concerning  the  development  of  the  engineering  description  package 
were  generally  supported  in  this  study.  The  evaluation  of  the  users 
guide  indicated  that  more  information  should  be  included  regarding 
the  location  of  and  access  to  the  various  component  parts  of  the 
system.  Recommendations  also  indicated  that  less  information  on 
general  theory  of  operation  need  be  presented.  Informal  feedback 
from  the  technician  estimators  also  indicated  that  technicians 
should  review  the  engineering  data  package  prior  to  field  data 
collection  to  ensure  that  all  technical  terms  are  either  understood 
by  the  technicians  or  adequately  defined  within  the  engineering 
description  package. 

The  level  of  detail  in  the  basic  engineering  description 
package  was  similar  to  engineering  description  packages  used  in 
previous  studies  (Sauer  &  Askren,  1978a;  Whalen  &  Askren,  1974).  A 
supplement  to  the  basic  engineering  description  package  was  prepared 
to  determine  the  effect  of  additional  engineering  data  on  the 
quality  of  the  estimates.  The  fact  that  the  additional  detail  had 
virtually  no  effect  on  the  estimates  suggests  that  there  is  a  point 
of  diminishing  returns  on  the  effort  spent  to  collect  all  possible 
details  of  the  new  system.  While  it  is  not  possible  to  state  pre¬ 
cisely  where  this  point  may  be  for  a  given  system,  the  engineering 
description  packages  used  in  this  and  the  previous  studies  can  be 
used  as  guides  to  the  level  of  engineering  detail  necessary.  It  was 
also  interesting  to  note  that  technicians  based  only  33  percent  of 
each  estimate  on  the  engineering  data  provided.  Although  no  similar 
previous  data  were  available  for  comparison,  these  percentages 
appear  to  be  reasonable.  Technicians  would  probably  rely  less  on 
the  engineering  data  and  more  on  their  past  maintenance  experience 
when  making  maintenance  manpower  estimates.  It  was  not  possible  to 
determine  any  relationships  between  amount  of  engineering  detail  and 
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the  degree  to  which  the  engineering  data  are  considered  in  the 
estimating  process.  The  utility  of  any  such  relationships,  however, 
would  be  questionable,  since  the  amount  of  engineering  detail  available 
in  the  early  design  stages  of  a  system  is  likely  to  be  very  limited. 

The  questionnaire  was  also  developed  following  the  recommen¬ 
dations  contained  in  the  prototype  users  guide.  The  resulting 
estimates  indicated  that  the  recommendations  were  generally  sound. 

The  superiority  of  the  on-equipment  person-hour  values  for  specific 
tasks,  for  example,  supported  the  recommendation  that  the  tasks  for 
which  person-hour  values  are  to  be  derived  be  described  in  as  much 
detail  as  possible. 

The  results  of  this  study  generally  reinforced  the  recommen¬ 
dations  with  regard  to  the  qualifications  of  the  estimators.  Tech¬ 
nicians  were  selected  from  AFSCs  within  the  career  field  most  closely 
related  to  the  new  system.  With  only  a  few  exceptions,  there  were 
no  differences  among  the  groups  in  terms  of  accuracy  of  their  esti¬ 
mates.  In  the  cases  where  differences  did  occur,  however,  there 
were  no  consistent  relationships  apparent  between  accurate  estimates 
and  group  characteristics.  This  evidence  suggests  that,  while  it 
may  be  desirable  to  collect  maintenance  manpower  estimates  from 
technicians  with  the  five-digit  AFSC  most  closely  matching  the- 
technology  of  the  proposed  system,  adequate  estimates  can  be  made 
using  technicians  from  the  general  career  field,  that  is,  the  first 
three  digits  of  the  AFSC  code.  This  finding  indicates  that  users  of 
the  expert  estimate  method  have  some  flexibility  in  selecting  the 
technicians  who  may  participate  as  estimators.  This  flexibility  of 
selection  would  be  most  advantageous  when  time  and  financial  resources 
are  limited. 

The  skill  level  recommendations  for  technicians  used  as  esti¬ 
mators  were  not  followed  exactly.  While  the  majority  of  the  estima¬ 
tors  (87  percent)  were  at  least  5-level  technicians  as  recommended, 
3-level  technicians  were  asked  to  participate.  Since  only  nine  3- 
level  technicians  participated,  however,  it  was  not  possible  to 
assess  adequately  the  quality  of  their  estimates.  Until  more  esti¬ 
mate  data  can  be  gathered  on  3-level  technicians,  the  recommendation 
that  at  least  5-level  technicians  be  used  as  estimators  should 
remain.  Support  for  this  statement  came  from  the  fact  that,  given 
an  opportunity  to  participate  as  estimators,  several  3-level  techni¬ 
cians  declined  to  participate  on  the  grounds  that  their  maintenance 
experience  was  too  limited. 

With  regard  to  the  quantity  of  estimators  (raters) ,  the 
prototype  guide  recommended  at  least  10  estimators  be  used.  How¬ 
ever,  if  resources  permit,  20  estimators  should  be  used  to  decrease 
the  amount  of  variability  and  to  increase  the  accuracy  of  the  esti¬ 
mated  data.  The  person-hour  cost  data  indicated  that  little  in 
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terms  of  person-hour  resources  would  be  saved  by  recommending  the 
minimum  of  10  estimators.  The  person-hour  costs  would  increase  only 
9  percent  when  increasing  the  number  of  estimators  from  10  to  25. 

It  would  seem  that  the  relatively  small  increase  in  cost  would  be 
offset  by  the  gains  in  accuracy  and  reduced  variability  (Sauer  & 
Askren,  i978a).  The  recommended  minimum  number  of  estimators  can, 
therefore,  be  increased  to  25,  while  incurring  a  relatively  small 
increase  in  overall  cost  of  the  method. 

Accuracy  of  the  Estimated  Data 

The  accuracy  of  the  estimated  maintenance  manpower  data  items 
were,  of  course,  the  ultimate  indicators  of  the  utility  of  the 
expert  estimate  method.  The  types  of  data  items  accurately  esti¬ 
mated  in  this  study  were  crew  size,  skill  level,  troubleshooting 
difficulty,  career  field/AFSC  requirements,  training  times,  and 
derived  estimates  of  maintenance  person-hours.  With  the  exception 
of  career  field  and  training  time  estimates,  the  types  of  data 
successfully  estimated  in  this  study  were  similar  to  those  suc¬ 
cessfully  estimated  in  the  Sauer  &  Askren  (1978a)  study. 

I  terns  which  were  not  accurately  estimated  include  percentage  of 
maintenance  task  occurrences,  support  equipment  requirements,  and 
derived  estimates  of  person-hours  for  troubleshooting  and  specific 
off-equipment  tasks.  In  the  case  of  the  troubleshooting  person-hour 
values,  the  poor  accuracy  scores  may  have  been  due  to  the  AFM  66-1 
data  rather  than  to  the  technicians'  inability  to  estimate  data  for 
the  derivation  of  these  person-hours.  The  AFM  66-1  person-hour  data 
for  troubleshooting  tasks  appeared  to  be  unusually  high,  the  result, 
perhaps,  of  inexperienced  maintenance  technicians  and  unfamiliar 
maintenance  requirements  of  the  new  system.  Troubleshooting  person- 
hour  data  reported  later  in  the  life  of  the  system  may  be  consider¬ 
ably  lower  and,  therefore,  closer  to  the  technicians'  estimated 
person-hours.  Person-hour  data  collected  2  or  3  years  from  the  date 
of  this  study  may  indicate  that  the  technicians'  estimates  were 
accurate  for  a  more  mature  system.  Similar  results  might  be  found 
for  the  specific  off-equipment  person-hour  values  and  the  percent  of 
maintenance  task  occurrence  estimates. 

The  poor  response  rate  and  poor  estimates  associated  with  the 
support  equipment  requirements  question  appeared  to  be  at  least 
partially  the  result  of  two  factors.  The  support  equipment  require¬ 
ments  question  was  placed  near  the  end  of  the  questionnaire.  Tech¬ 
nicians  reached  this  point  after  spending  approximately  1  1/2  hours 
on  the  questionnaire  and  may  have  begun  to  experience  boredom  or 
fatigue.  Aggravating  this  potential  fatigue  factor  was  the  fact 
that  the  question  was  written  in  an  open-end  response  format.  The 


technician  not  only  had  to  organize  a  potentially  lengthy  response 
but  was  not  afforded  any  types  of  response  cues  inherent  in  a 
checklist  or  multiple-choice  format.  A  change  of  format  and  physi¬ 
cal  location  within  the  questionnaire  may  improve  both  the  response 
rate  and  the  quality  of  the  responses  regarding  support  equipment 
requirements. 

Cost  Data 


A  direct  comparison  of  the  cost  of  the  expert  estimate  method 
with  another  data-generating  method  was  not  possible.  Aside  from 
the  fact  that  such  a  comparison  was  outside  the  scope  of  the  present 
study,  a  comparable  data-generating  method  intended  for  use  in  the 
early  design  phase  of  a  system  could  not  be  identified.  However, 
informal  assessments  from  professionals  experienced  in  the  collec¬ 
tion  of  manpower  data  indicated  that  the  expert  estimate  method 
appears  to  be  a  low  cost  and  relatively  rapid  method  for  collecting 
these  data.  It  was  possible  to  compare  the  costs  of  the  various 
activities  which  comprise  the  expert  estimate  method.  The  recom¬ 
mendation  to  use  a  larger  number  of  estimators  was  based  on  such  an 
analysis.  However,  until  a  comparison  can  be  made  with  another 
data-generating  method,  the  cost  data  simply  stand  as  a  cost  record 
of  one  particular  application  of  the  expert  estimate  method. 

Evaluation  and  Revision  of  Prototype  Users  Guide 

Another  objective  of  this  study  was  to  revise  the  prototype 
users  guide  based  on  the  experience  gained  from  the  application  of 
the  expert  estimate  method  and  on  a  critique  of  the  guide  itself . 

The  comments  generated  by  the  critique  were  valuable  in  identifying 
those  areas  of  the  guide  which  needed  clarification,  where  examples 
could  be  used  more  advantageously,  and  what  types  of  information 
should  be  included  in  the  engineering  description  package.  These 
comments  were  taken  into  account  when  revising  the  guide.  Comments 
also  indicated  that  the  method,  with  further  development,  would  be 
best  applied  at  the  Milestone  0  and  Milestone  1  decisions  in  the 
Defense  Systems  Acquisition  Review  Council  process.  The  most 
valuable  comment,  however,  identified  the  direction  in  which  this 
further  development  of  the  expert  estimate  method  should  proceed. 
Research  and  development  must  be  directed  toward  producing  a  system- 
level  estimate  of  maintenance  and  manpower  requirements.  Although 
some  of  the  estimates,  such  as  training  time  and  career  field/AFSC, 
already  represent  system-level  estimates,  the  task  time,  person- 
hour,  crew  size,  and  skill  level  estimates  are  made  at  the  component 
or  line  replaceable  unit  level.  Some  method  needs  to  be  developed 
to  combine  these  component  estimates  in  such  a  way  that  systems- 
level  estimates  can  be  produced.  It  should  be  noted  that  the  method 
is  useful  in  its  present  form  for  trade-off  comparisons  which  do  not 
need  to  aggregate  to  a  full  system  estimate.  However,  with  this 
type  of  development,  the  expert  estimate  method  can  become  a  more 
versatile  tool  in  the  early  design  stages  of  new  systems. 


CONCLUSIONS  AND  RECOMMENDATIONS 


The  following  conclusions  and  recommendations  are  made  based  on 

the  findings  of  this  study. 

1.  The  expert  estimate  method  as  documented  in  the  prototype  users 
guide  (Sauer  &  Askren,  1978b)  was  successfully  applied  to  a 
ground-based  radar  system  to  generate  estimates  of  maintenance, 
manpower,  and  other  types  of  data. 

2.  Most  of  the  procedures  and  recommendations  for  applying  the 
expert  estimate  method  were  supported  by  both  the  tryout  and 
the  critique  of  the  prototype  guide.  Changes  that  were  made  to 
the  users  guide  included  the  adoption  of  a  larger  recommended 
minimum  number  of  estimators  (25)  and  additional  guidance 
regarding  preparation  of  the  questionnaire. 

3.  This  application  indicated  that  the  expert  estimate  method  can 
be  used  to  generate  estimates  of  crew  size,  skill  level,  career 
field  and  Air  Force  Specialty  Code,  troubleshooting  difficulty, 
training  time,  and  derived  estimates  of  maintenance  person- 
hours. 

4.  The  expert  estimate  method  has  not  generated  accurate  estimates 

of:  percent  of  maintenance  task  occurrences,  support  equipment 

requirements,  and  derived  estimates  of  person-hours  for  trouble¬ 
shooting  and  specific  off-equipment  maintenance  tasks. 

5.  Technicians  indicated  that  33  percent  of  any  given  estimate  was 
based  on  information  in  the  engineering  description  package, 
while  67  percent  of  any  given  estimate  was  based  on  their 
maintenance  experience. 

6.  The  expert  estimate  method  appears  to  be  a  relatively  inexpen¬ 
sive  and  rapid  method  for  generating  maintenance  manpower  data 

in  the  early  stages  of  system  design.  No  similar  data-generating 
methods,  however,  were  found  for  comparison  purposes. 

7.  As  an  inexpensive  means  to  gain  more  information  on  the  accu¬ 
racy  of  the  estimated  data,  AFM  66-1  maintenance  data  for  the 
AN/TPS-43(E)  radar  system  should  continue  to  be  collected. 

Twelve  months  of  data  collected  on  either  an  annual  or  semi¬ 
annual  basis  could  be  used  to  determine  how  the  accuracy  of  the 
estimates  changes  as  more  maintenance  experience  is  gained  with 
the  system. 

8.  Research  needs  to  be  directed  toward  identifying  a  method  of 
combining  the  estimated  data  at  the  line  replaceable  unit  level 
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into  system-level  estimates.  Relevant  data  items  would  include 
task  times,  crew  size,  skill  level,  and  person-hours. 

9.  The  expert  estimate  method  should  now  be  applied  to  an  early 

design  stage  system  analysis  to  gain  further  insights  into  the 
utility,  general  applicability,  and  cost-effectiveness  of  the 
method.  The  validity  of  the  method  could  also  be  evaluated 
when  operational  data  for  the  system  become  available  and  can 
then  be  compared  to  the  early  design  stage  estimates. 
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GLOSSARY 


AFTO  Form  349 — The  title  of  this  form  is  the  "Maintenance  Data 

Collection  Record."  It  is  used  to  record  maintenance  actions 
on  various  types  of  equipment.  The  data  which  may  be  recorded 
include:  job  identification  data,  component  identification 

data,  maintenance  task  time,  crew  size,  type  of  maintenance, 
malfunctions,  discrepancies,  and  specific  maintenance  actions 
taken. 

Air  Force  Regulation  39-1 — The  Airman  Classification  Regulation 

defines  and  describes  all  Air  Force  occupational  specialties 
and  career  fields  and  their  training  and  skill  requirements. 

Air  Force  Manual  66-1 — This  manual,  "Maintenance  management  policy," 
establishes  the  maintenance  management  system  applicable  to  all 
Air  Force  activities  engaged  in  the  maintenance  of  aircraft, 
missiles,  munitions,  aerospace  ground  support  equipment, 
avionics,  training  equipment,  and  communications-electronics- 
meteorological  equipment. 

Air  Force  Specialty  Code — The  Air  Force  Specialty  Code  (AFSC)  is  a 
five-digit  number  assigned  to  Air  Force  enlisted  personnel 
which  defines  the  general  career  field  and  the  specialty  within 
the  career  field  in  which  the  individual  is  qualified  and  the 
skill  level  the  individual  has  attained. 

Career  Field — Career  Field  represents  a  general  area  of  expertise  or 
technology  in  which  an  individual  is  qualified.  The  first 
three  digits  of  the  Air  Force  Specialty  Code  (AFSC)  specify  the 
individual's  career  field.  For  example,  the  career  field  of 
avionic  weapon  delivery  systems  (general  area)  will  include 
three  specialty  areas:  bomb-navigation  systems,  defensive  fire 
control  systems,  and  weapon  control  systems. 

Field  Training  Detachment  Training — Field  Training  Detachments 

provide  job-oriented  maintenance  training  on  assigned  weapon 
systems,  support  systems,  and  selective  equipment.  The  FTDs 
require  host  support  in  order  to  provide  responsive  and  quality 
training. 

Off-Equipment  Maintenance — Maintenance  activities  which  cannot  be 
accomplished  on  the  system.  Maintenance  must  be  accomplished 
in  the  maintenance  shop. 

On-Equipment  Maintenance--Maintenance  actions  which  may  be  performed 
on  the  system.  No  shop  maintenance  is  required. 
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On-The-Job  Training — Training  which  is  accomplished  in  the  course  of 
normal  operations  or  maintenance  activities.  The  trainee 
learns  by  observing  and/or  actually  performing  real  operations 
or  maintenance  tasks. 

Skill  Level — Skill  level  represents  the  level  of  qualification  and 

degree  of  expertise  achieved  within  a  technician's  career  field 
and  specialty  code.  The  five  skill  level  codes  are  1,  3,  5,  7, 
and  9,  with  skill  level  1  equivalent  to  a  helper,  skill  level  3 
equivalent  to  an  apprentice,  skill  level  5  equivalent  to  a 
specialist,  skill  level  7  equivalent  to  a  technician,  and  skill 
level  9  equivalent  to  a  superintendent.  This  code  appears  as 
the  fourth  digit  of  an  individual's  Air  Force  Specialty  Code 
(AFSC) . 

Technical  Training — Formal  classroom  training  conducted  away  from 
the  job  site  at  designated  Technical  Training  Centers. 

Work  Unit  Code — The  work  unit  code  is  a  five-character  code  used  to 
identify  systems,  subsystems,  and  components  for  which  mainte¬ 
nance  is  required  or  on  which  maintenance  was  accomplished. 


